A Multi-Scale Electromagnetic Particle Code with Adaptive Mesh Refinement and Its Parallelization

To investigate multi-scale phenomena in space plasma including plasma kinetic effects, we started to develop a new electromagnetic Particle-In-Cell (PIC) code with Adaptive Mesh Refinement (AMR) technique. In AMR simulation, spatial grid size and time step intervals are defined according to the hierarchy levels, where high and low levels correspond to the fine and coarse grid systems, respectively. As the simulation system evolves, some complex micro-scale phenomena can locally and intermittently occur in a hierarchical domain (Level L). If the grid size in Level L is too coarse to simulate the local complex phenomena, A higher hierarchical domain (Level L+1) is adaptively created in which the grid spacing size and the time step interval become half of those used in the domain of (Level L). AMR-PIC simulations require smaller amount of memory and shorter computation time than conventional PIC simulations. In parallelizing the code for a distributed many-core system, we adopt a scheme of dynamic domain decomposition which realizes the load balance between distributed subdomains. In conventional domain decomposition scheme, one of the issues is load balance between processors because the number of spatial grids and particles belonging to each subdomain is not always constant due to the AMR procedure. Since the cost of particle calculation is dominant, which is about 70-80% of the total cost, the number of particle calculation loops should be balanced between each processor. For this purpose we are introducing dynamic domain decomposition called DDD. In DDD, we calculate the work weights based on the cost of particle calculation and hierarchical levels of each cell. Then we decompose the domain according to the Morton curve and the work weight, so that each processor has approximately the same amount of work. By performing a simple onedimensional simulation, we confirmed that the dynamic load balancing is achieved and the computation time is reduced by introducing the dynamic domain decomposition scheme. We are currently incorporating this scheme in the three dimensional version of AMR-PIC code and are performing a test simulation to examine the parallelization efficiency as well as the scalability with respect to the number of processors.